Method and system for collecting and organizing data corresponding to an event

ABSTRACT

A system and method for analyzing data from a plurality of computer environments. The computer environments are authenticated and data is imported to a memory location. The data is converted into a uniform format to enable expedited searching by one or more authenticated users. The data may be marked so that a user may determine which computer environment provided the data. The system may also create one or more indexes of the data to assist one or more users in searching the data.

BACKGROUND OF THE INVENTION

As more businesses and governmental entities increasingly rely oncomputer networks to conduct their operations and store relevant data,security of these networks has become increasingly important. The needfor increased security is emphasized when these networks are connectedto non-secure networks such as the Internet. The preservation ofimportant data and the ability to retrieve the data in the aftermath ofa security breach has become the focus of network administrators.

Another concern for network administrators is the retrieval, searching,sorting, and copying of relevant electronic data for use in legalprocedures, such as responses to subpoenas. Many companies, especiallylarge companies, routinely receive legal document requests as part ofcivil or criminal proceedings. Fulfilling the company's duty to respondto these requests typically requires the company to search not onlytheir paper documents, but their electronic data such as e-mail, wordprocessing files, spreadsheets, databases, and images as well. Thisprocess of gathering, searching, sorting, and copying electronic datawithout damaging the original data can be extremely expensive andtime-consuming.

Electronic data, such as word processing documents, may also includemetadata, which is information about the electronic data such ascreation date and time or revision history. Searching, sorting, andcopying of metadata presents an additional challenge to companiespreparing to respond to legal process or recover from a security breach.

Various technologies may be employed to aid in the processing andclassification of data, including search technologies, software thatcopies the entire contents of the hard drive in a computer system, andsoftware that allows an analyst to review its contents and categorize itbased on their observations. But existing solutions address only subsetsof the problem, such as collection, analysis, or reporting, and fail tofully encompass the process of collection, analysis/minimization, andreporting. Thus, network administrators must employ multipletechnologies at each phase and sometimes manually handle the transfer ofdata between the phases. For example, one technology might be used toimage a system hard drive or obtain a process list, while an entirelydifferent technology is used to review that information. This results inincreased costs and time delays, which are undesirable in both dataproduction and security breach recovery processes.

Thus there exists a need for additional technologies to manage theentire data production and security breach recovery processes in afashion that controls costs and reduces risk.

SUMMARY OF THE INVENTION

In response to this need, the present application describes a method andsystem for data preservation and retrieval. A method according to thepresent invention provides for the identification, collection, analysisand reporting of information necessary to resolve a computer securityincident or data request incident. The method and system employtechniques to model and organize data such that it allows for precisionacquisition of information to resolve an incident, analysis of that dataafter acquisition, and reporting on conclusions reached during analysis.Furthermore, methods are provided to support collaboration across eachphase of the process.

In accordance with an embodiment of the invention, data can be organizedand presented in response to an event comprising authenticating aplurality of memory sources, importing stored data from a plurality ofmemory sources into a first memory, converting the stored data into aspecified format to produce uniform data, providing an interface to auser through which the user may access the uniform data, and presentingone or more subsets of the uniform data to the user in response to arequest from the user.

The plurality of memory sources can be coupled to the first memorythrough a network such as, for example, the Internet. The plurality ofmemory sources can be servers or disk drives. The specified format canbe extensible markup language or plain text format.

The user can be allowed to search the uniform data through theinterface. The interface can be provided on a computer coupled to anetwork and the first memory is also coupled to the network. The uniformdata can be stored in a database. The stored data can be both contentand metadata.

The interface can be provided to a plurality of users through a networksuch as, for example, the Internet.

In accordance with another embodiment of the invention, stored data canbe analyzed by searching for a plurality of storage devices coupled to anetwork, copying the stored data on the plurality of storage devices toa memory location, converting the stored data to uniform data comprisinga single format, providing an interface to a user through which the usermay access the uniform data, and presenting one or more subsets of theuniform data to the user in response to a request from the user.

The plurality of storage devices can be servers or disk drives. Thememory location can be a disk drive. The single format can be extensiblemarkup language.

The user can be presented with a list of the plurality of storagedevices coupled to the network and allowing the user to select which ofthe plurality of storage devices to copy stored data from.

The user can be presented with a list of data types present on theplurality of storage devices coupled to the network and allowing theuser to select which of the data types to copy. The list of data typescan be content and metadata.

The single format data can be added to a database. The user can beprovided with access to the database through an interface.

In accordance with another embodiment of the invention, a system foranalyzing data can include a means for searching a network for storagedevices, means for identifying the storage devices to a user, means forallowing the user to choose selected storage devices, means for copyingthe data from the selected storage devices, and means for converting thedata to a converted data having a single format, means for providingaccess to the user to the converted data for analysis.

In accordance with an embodiment of the invention data can be searchedusing one or more indexes comprising identifying a plurality of computerenvironments coupled to a network, searching the plurality of computerenvironments for data that matches a first predetermined criterion,copying the matching data to a memory location, converting the matchingdata to a uniform format, creating one or more indexes of the matchingdata, and providing an interface to a user that allows the user tosearch the matching data using the one or more indexes.

The creating one or more indexes of the matching data can be creatingone or more search documents containing information about the matchingdata. The one or more search documents can be created in a predeterminedformat. The search documents can be stored in a database.

In an embodiment, creating one or more indexes of the matching data caninclude searching the matching data for particular term and creating asearch document including a list of documents containing the particularterm. The list of documents containing the particular term can includethe storage location of those documents. The one or more indexes canrank a plurality of particular terms based on the relevance of theparticular terms.

In an embodiment, creating one or more indexes can include creating anindex of content and an index of metadata or indexing only apredetermined portion of a document. The predetermined portion can bethe title.

In accordance with an embodiment, the creating one or more indexes caninclude indexing only a predetermined file type.

Documents can be ranked in an index based on predetermined criteria. Thepredetermined criteria can be the number of times a particular documenthas been viewed.

In accordance with another embodiment of the invention, files can besearched by identifying a plurality of computer environments coupled toa network, searching the plurality of computer environments for filesthat match a predetermined criteria, converting matching files to apredetermined format, creating one or more indexes of the matchingfiles, and providing an interface to a user allowing the user to searchthe matching files using the one or more indexes. The predeterminedcriteria can be a file type.

The one or more indexes can include a content index or a metadata index.An index can contain each word included in each file. The index caninclude predetermined terms. The index can include each file containingpredetermined terms and the storage location of that file.

The creating of one or more indexes of matching files can includecreating a search document for each matching file containing data abouteach matching file. The terms can be ranked in an index.

In accordance with another embodiment of the invention, a system forsearching data can include a means for identifying a plurality ofcomputer environments coupled to a network, means for searching theplurality of computer environments for data, means for copying the datato a memory location, means for creating one or more indexes of thedata, and means for searching the data using the one or more indexes.

In accordance with an embodiment of the invention, data can be collectedby identifying a plurality of computer environments coupled to anetwork, searching the plurality of computer environments for data thatmatches a first predetermined criterion, copying matching data to amemory location, creating one or more indexes of the matching data, andupdating the indexes at a predetermined interval. The predeterminedinterval can be a time interval.

The updating of the indexes can include creating one or more new indexesbased on recently discovered data, deleting one or more indexes based onrecently deleted data, or creating one or more new indexes based on oneor more searches requested by a user. The indexes may be updated bydeleting one or more indexes based on one or more searches requested bythe user. The deleting of one or more indexes based on one or moresearches requested by the user can include deleting one or more indexesif the indexes have not been utilized by the system within apredetermined period of time.

The updating of the indexes at a predetermined interval can includemerging two or more indexes.

In one embodiment, the updating of the indexes at a predeterminedinterval comprises creating a temporary index for a new amount of datawhen the new amount of data has been detected during the searching ofthe plurality of computer environments. Further, the temporary index canbe merged into a second index once the temporary index has reached apredetermined size. The temporary index can be merged into a secondindex at a predetermined interval, such as a predetermined timeinterval.

In accordance with another embodiment of the invention, data in acomputer environment can be searched by presenting a list of storagedevices to a user, allowing the user to select storage devices,searching for stored data on the selected storage devices, creating aplurality of indexes of the stored data, and executing a search of thestored data based on the indexes. The user can be allowed to define acriteria and searching only for stored data on the selected storagedevices that meet the criteria. The stored data can be copied to asingle memory location.

The plurality of indexes can be tables which include a number ofvariables and the files in the stored data that correspond to thevariables. The variables can be file types or file creation dates. Thetables can further include the location of the stored data on theselected storage devices. The indexes can be saved to a storage device.

In an embodiment, the indexes are periodically updated upon thediscovery of a predetermined amount of unindexed data.

In accordance with another embodiment of the invention a system forcollecting data can include a means for identifying a plurality ofcomputer environments coupled to a network, means for searching theplurality of computer environments for data that matches a firstpredetermined criterion, means for copying the matching data to a memorylocation, means for creating one or more indexes of the matching data,and means for updating the indexes at a predetermined interval.

The present invention discloses systems and methods for the preparationfor an event, the initiation of a response to the event, collection ofdata, analysis of the data, organization and presentation of data, andresolution of the event.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 depicts various components of the invention in one embodiment.

FIG. 2 depicts various components of an agent according to oneembodiment of the present invention.

FIG. 3 depicts various components of a controller according to oneembodiment of the present invention.

FIGS. 4-7 are flow diagrams which depict interaction between componentsof the invention.

FIGS. 8-9 is a flow diagram showing a method according to one embodimentof the present invention.

FIG. 10 depicts the translation of data into a uniform representationaccording to one embodiment of the invention.

FIG. 11 depicts a document to be used with a system according to oneembodiment of the present invention.

FIGS. 12-19 are flow diagrams depicting the steps of a method accordingto the present invention.

FIGS. 20-22 depict results presented to a user of analyses conducted bya system according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present application describes a method and system for datapreservation and retrieval. A method according to the present inventionincludes a number of steps. The initial step is preparation, whichencompasses measures taken before an incident such as a request forproduction of data or a security breach occurs to ensure that theresources for responding to the incident have been identified, theassets such as data storage devices or networks have been identified,and potential actions to be taken in response to the incident have beenoutlined. The second step is initiation in which the incident isidentified and the process is initiated. The third step is collection inwhich information such as data and metadata are collected. The data tobe collected and the sources of the data must be identified. In the caseof legal process, the data to be collected may be identified in a legaldocument such as a subpoena. The fourth step is the analysis of thedata. This analysis may include sorting, searching, reviewing, andorganizing the data. In the case of a security breach, the method usedto conduct the breach must be determined and any damage caused by thebreach must be evaluated. In the case of generating a response to legalprocedures, all data that is responsive to the legal procedures must beidentified and collected, while confidential data and non-responsivedata must be segregated. The fifth step is presentation, in which thefindings are presented to a user. In the case of a security breach,details of the security breach and identification of the damage arepresented to a user. In the case of a response to legal process, thegathered information is presented to a user. The final step isresolution, in which the cause of any security breach is corrected toprevent future breaches or the user to which the documents are presentedverifies that the project is complete.

Data gathered for security incident response and response to legalprocedures may take many forms and may be gathered for many reasons.Documents and e-mail messages may be gathered to assess informationsought by an attacker during a security breach, determine how a breachoccurred, or determine whether an attacker has stored data on a system.Documents and e-mail messages may also be gathered as responsive tolegal requests. Data concerning deleted files on certain storage devicesmay also be relevant to show deletions caused during a security breachor deletions of relevant material that must be retrieved to respond tolegal process. Programs or processes used on a computer system may alsobe relevant to identify in response to legal process or to identify thesource of particular relevant documents. Programs or processes may alsodispel or confirm a suspicion that a security breach has occurred. Anattacker may also install programs or processes that must be removed.Configuration data of a computer system or network may also beresponsive to legal process or identify changes that an attacker hasmade during a security breach. Configuration data may also identifyrelevant components of a system that may contain data responsive tolegal process, such as network drives or what types of information arestored on a system. Databases are important to reveal information thatmay have been accessed during a security breach, such as customerinformation or financial information. Databases may also containrelevant information that may be responsive to legal process.

The data necessary to respond to a security breach or legal process ispotentially voluminous and takes many forms. This data may also belocated on various different storage devices, which may make retrievalmore difficult.

Once the data has been acquired, it must be analyzed to find therelevant pieces that confirm or dispel a security breach or answer alegal discovery request. There are many different techniques which maybe used to analyze the data. Keyword searching may be used to finddocuments relevant to a particular topic in response to legal process orpatterns of files relating to a particular topic. Certain keywords orbyte patterns may be searched as indicators of compromise. For example,if an investigator identifies a specific module of malware and findsthat a specific byte sequence occurs within memory when it is running,or within a file that contains the malware, searching for that bytesequence may be an important step in evaluating a security breach.

Various methods may be used to search for specific files in a computersystem's memory or on its hard drive, including examination of portionsof file contents or calculation of hash values. A hash algorithm mayalso be used to search for encrypted files on computer systems in thefollowing manner: a hash algorithm is run on a file to generate anumber. The hash algorithm is then run on each file in a data set andthe numbers are compared to the number generated by the file. If anyfile produces the same numerical result as the number generated by thefile, it is reasonably certain that they are matching files.

Indexed searching may be used to analyze the data. Indexed searchingrefers to the use of search engine technology to classify the contentsof a set of data by creating special references called indexes in a setof data. The indexes make it possible to execute fast lookups ofinformation that may be responsive to legal process or compromised froma security breach.

The concept of a union may also be used in response to a security breachor legal process. In the case of a security breach, a union is thecollection of all of the indicators of compromise from each individualattack form. In response to legal process, the union describes all ofthe forms of data being gathered in response to a single request. Thefinal data set provided in response to the legal process is the union ofall minimized data found to be relevant during an acquisition.

The concept of an intersection may also be used for data analysis.Intersection of data occurs when data that is present on more than onestorage devices. Intersections commonly occur on networks when the samefiles are stored on more than one network drive. Intersections of datamay be analyzed to determine whether data has been damaged as a resultof a security breach. Intersections may also be used in response tolegal process to show information about files, such as access to files,creation dates of files, and revision of files. For example, anintersection analysis may reveal all files with the same creation date,the list of files that are common across multiple computer systems, orthe list of recipients that are common across a set of e-mail. Analysisof intersections may also prevent the copying and presentation ofduplicate files.

Difference analysis may also be used which compares data sets stored ondifferent devices or computer systems. Difference analysis may be usedto compare a computer system that has not been compromised with anothersystem to determine whether a compromise has occurred. These differencescan be calculated across a wide set of data, including file lists,process listings, system memory state, network connections, and so on.Difference analysis may also be used in generating a response to legalprocess by eliminating redundancies between computer systems ordetermining which files on a particular computer system have beenchanged over time.

A timeline analysis may also be conducted to determine how a computersystem has changed over time. This is helpful in finding indicators of asecurity breach or determining exactly what has changed when a securitybreach has occurred so that the changes may be corrected. For example,during a computer security incident, looking at file modification datesin line with the dates/times for various system log entries can helpcreate a picture of the activity an attacker may have engaged in whileon a system. Timeline analysis is also used in responding to legalprocess by including or excluding data that was created during arelevant time period. Data may be normalized to compare differentclasses of information in the same context.

Data normalization and transformation may occur during the analysisphase. Normalization is the process of “lining up” two disparate piecesof information based on their common fields. It may be necessary tochange the representation, structure, or content of investigative datain order to perform certain types of analyses. For example, in asituation where data containing time stamps is acquired from a computersystem, if the clock on the target computer system was off by someamount, comparing the time data from that system against time data fromother systems may be difficult. If the amount of time “skew” on thetarget computer system is known, it is possible to transform thetimestamps on all data acquired from that system to some standard—if theclock was 2 minutes fast, 2 minutes could be subtracted from all timestamp values in order to normalize them for comparison to other systems.

During the analysis, it may be desirable to annotate or markup certainpieces of data. This may make it easier to refer back to findings,collaborate with other investigators, and eventually write reports.

The software that executes the various operations of the presentinvention may reside on many different hardware configurations whichwould be known to one skilled in the art. One such configurationinvolves a multi-tiered approach with software components on separatecomputer systems as depicted in FIG. 1. In one embodiment, a system andmethod according to the present invention is divided into three primarycomponents: a data collector (called an agent); middleware for storage,management, and analysis of data (called a controller); and a userinterface that allows end-users to perform collection, analysis, andreporting operations in a collaborative environment (called a console).

An agent is a module of software installed on a target system 20 thatenables a user to monitor and interact with the target system 20. Agentsallow users to gather information about multiple aspects of the targetsystem 20. Agents also permit users to remotely retrieve the contents ofthe target system's memory or hard drive, and could potentially beconfigured to modify its contents. The agent may be configured to eithercommunicate over a computer network, or to read and write all relevantconfiguration information and acquired data to a computer storagemedium, such as a hard drive or removable read/write media (USB key,etc). In one embodiment, the agent is built in a modular fashion. Theability to gather a particular piece of data from a target system 20(e.g. a list of running processes on the target system) is implementedas a discrete module of software and loaded by the agent. This allowsfor easy adaptation of the agent to different environments that havespecific requirements for data collection.

One embodiment of the various software components contained in the agentis depicted in FIG. 2. The agent 200 preferably has at least one auditor210 and at least one imager 220. A series of application programminginterfaces (APIs) built on those services provide an extensibilityframework 230, making it cost effective to implement individual modules.Auditor 210 and imager 220 modules can then be built on top of theseservices. In this embodiment, shared features and services such ascommunication and security are built into a foundation layer of software240. Auditors are software modules that obtain different kinds ofinformation from the system, and typically revolve around currentexecution and data states. Examples of auditor modules include: processlistings, system configuration settings, network connection statelistings, and file listings. Imagers obtain verbatim copies of data fromthe system. Examples of imager modules include: disk acquisition, memoryacquisition, and file acquisition. Additional module types could also bedefined, such as Workers, which could modify the state of a runningsystem by altering either information stored on a hard drive or inmemory.

Referring back to FIG. 1, a controller 30 is coupled to target system 20through a network. The controller 30 comprises software installed on oneor more computer systems whose function is to interact with targetsystems 20 for purposes of gathering information from target systems 20,and the interface software provided for users to interact with theproduct. Controllers 30 connect and communicate with target systems 20and interfaces over a computer network. Analysis of acquired data isconducted centrally by the controller, allowing users to workcollaboratively on the same set of data. In one embodiment thecontroller 30 has a modular component for conducting analysis.

FIG. 3 illustrates the various components that may be present in acontroller in one embodiment of the invention. The controller 300 mayhave a core set of services and capabilities 380 that are implemented asa series of interacting software modules. These services providecapabilities for process intercommunication, interaction with connectedinterfaces, issuing data collection jobs to target systems andretrieving results, organizing and storing acquired data, and performinganalysis. There may be modules for conducting storage services 330,search and metadata services 340, analyses 350, and job management 360to manage workflow 370. On top of this structure an extensibility layer320 is implemented that allows analyzers 310 to interact with thecomputer system. An analyzer 310 is a software module that uses one ormore of the various services on the controller to manipulate one or moresets of data acquired from agents at the request of the user. Analyzersare typically some form of data evaluation or transformation module thatallows an end user to more effectively minimize data and identify itemsthey are looking for. Analyzers may be created to execute any one of theanalysis techniques discussed herein.

Multiple controllers can interact with one another to provide additionalprocessing power, storage space, and geographic proximity based on userrequirements. This kind of interconnection of multiple controllers iscalled a “cluster.” For example, if a user wanted to be able to analyzelarger amounts of data in less time, additional controllers could beadded, which would increase analysis speeds. If a user had a large,diverse network and wanted to position a controller “close” to a set ofsystems from which the user would be gathering a large amount of data(copying the contents of a hard drive, for example), a controller couldbe added to a cluster in order to prevent and manage network congestion.When additional controllers are added, the end user is still able tointeract with data stored on any controller within the cluster.

Referring back to FIG. 1, the console 40 provides the user with aninterface for using the system in a collaborative environment, providingmethods to execute jobs that acquire or analyze data, display acquireddata, mark up data, and author reports. Any additional featuresimplemented within the software are accessible by the user through thisinterface. The console 40 is coupled to one or more controllers 30 overa computer network and requests data from it necessary to render theinterface for the end user. All requests by the user to acquire newdata, analyze data, or modify data (e.g. mark up data, author reports,etc) are sent from the console 40 to the controller 30 for fulfillment.

Although FIG. 1 depicts one embodiment of various elements of theinvention coupled together through a network, one skilled in the artwould recognize that the components such as the target system and thecontrollers may be directly connected or even reside on the samehardware. The software necessary to implement the invention may functionon a number of different hardware devices, such as personal computers,servers, and portable electronic devices. In one embodiment, the targetsystem, controller, and console may all reside on a single computerwhich is not connected to any network. In another embodiment, the targetsystem may be a portable computing device and the controller may be aserver. In this embodiment, a user may use a personal computer tocommunicate with the server over the Internet to obtain and analyze datastored on the portable computing device.

The console exposes collaboration features to the user, providingmethods for users to work on the same data stored on a controller. Inone embodiment, multiple users should be able to view, organize,annotate, add, and modify data on a controller or cluster of controllersat the same time.

The one or more controllers may be configured with information aboutagents and the target systems they reside on. The controller must havethe necessary network address information to contact a target system andit must be able to uniquely identify the agent software running on thattarget computer. If the controller can not uniquely identify the agentit is communicating with, then it will not able to correctly organizethe data it collects from that agent, making it impossible to identifyfor an end user where acquired data originally came from.

In order to address these issues, one embodiment of the inventionprovides two methods for identifying network configuration data foragents: an automatic method called agent discovery and a manual methodwhere the end-user specifies all the necessary configuration data. Agentdiscovery is a service that is initiated over a computer network by anagent when the agent runs on a target computer. The agent is configuredwith the location of an agent discovery service when it is firstinstalled on the target system. This configuration is typically thenetwork address or name for the system where the agent discovery serviceis running. The agent authenticates itself to the agent discoveryservice and transmits information about itself to the service, includingnetwork configuration information. The agent will contact the serviceperiodically to update its configuration information; this ensures thedata remains up-to-date, even if it changes.

A controller may periodically contact the agent discovery service toidentify any new agents that have become active. When it receives thisinformation, it records the presence of that agent and its identity.Later, when a controller wishes to contact an agent, it asks the agentdiscovery service for the current network configuration information forthat agent. It then uses that information in order to connect to thatagent over a network.

FIG. 4 is a flow diagram depicting one example of the interactionsbetween an agent, agent discovery service, and a controller according toone embodiment of the invention. An Agent 400 installed on a target hostwith configuration for connecting to the Agent Discovery Service 405 andto permit connections from Controller 410 in a step 415. The AgentDiscovery Service 405 is configured to allow connection with Agents 400in a step 420. The Controller 410 is configured to allow connection toAgents 400 and Agent Discovery Services 405 in a step 425. Upon startup,Agent 400 connects to Agent Discovery Service 405, authenticates theAgent Discovery Service 405 to ensure it is within the proper trustdomain and transmits its network configuration in a step 430. The AgentDiscovery Service 405 accepts connection from the Agent 400 andregisters its credentials and network configuration in a step 440. TheAgent Discovery Service 405 identifies the list of Agents 400 that arenew since the last time the Controller 410 asked and provides thatinformation in a step 445. The Controller 410 periodically polls AgentDiscovery Service 405 to identify any new Agents 400 registered withAgent Discovery Service 405 in a step 450. The Agent 400 periodicallycontacts Agent Discovery Service 405 to update its network informationin a step 455. Agent Discovery Service 405 accepts connection from theAgent 400, looks up the Agent's record according to its credentials andupdates the network information in a step 460. The Controller 410creates new records for the newly discovered Agents so the user may rundata acquisition jobs against them in a step 465. The Controller 410receives a request from a user to run a job on an Agent 400 and pollsAgent Discovery Service 405 to request the updated network data for theAgent 400 in a step 475. The Agent Discovery Service 405 looks up thenetwork information for the requested Agent 400 and returns it to theController 410 in a step 470. The Controller 410 uses the networkinformation provided by Agent Discovery Service 405 to contact theselected Agent 400 and execute the job in a step 485. The Agent 400authenticates the Controller 410, receives and executes commands, andreturns the requested data in a step 480.

Optionally, a user may utilize a manual discovery process by connectingto a controller with a console and creating a record for an agent andtarget computer. This new record may then be configured manually by theuser via the console with the appropriate network settings so thecontroller can contact the invention. Required settings may include thenetwork address, port, and protocol that the controller must use tosuccessfully connect to that agent.

In addition to the methods for specifying network configurationdescribed above, one embodiment of the invention implements a securityprocess that provides for authentication of controllers and agents, andencryption of data transmitted between the components of the inventionfor added security. Public Key Infrastructure (PKI) technologies may beused to create the necessary components to manage aspects ofauthentication, authorization, and encryption within the system.

In one embodiment, a trust domain (TD) is maintained by creation of acertificate authority (CA), which is subsequently used to issuecredentials to participating entities and to validate that an entity isa member of a given trust domain. Generally speaking, a trust domaindefines a set of entities that are authorized to interoperate with oneanother on some level. The trust domain is defined by the collection oftrust domain entities and subscribing entities that are currentlyparticipating. Trust domain entities (TDE) include controllers and agentdiscovery service providers; they are typically entities that performsome service that requires them to be identified and authenticated toother entities. Subscribing entities (SE) are those that need toauthenticate an entity within the trust domain, and may include the enduser (through the console) and agents. Each TDE is also an SE forcertain operations. For example, when a controller is operating in acluster it needs to authenticate other controllers within that clusterto see if those controllers are authorized to communicate with it. Whena controller is authenticating another controller, it is an SE of thetrust domain.

An overview of the TDE and SE roles for components of the invention andthe end user is as follows. A controller must be authenticated when theyconnect to other controllers, agents, and agent discovery services in atrust domain entity role. Controllers authenticate to each other whenoperating in a cluster. Agents must authenticate controllers beforeaccepting commands for acquisitions. An agent discovery service mustauthenticate a controller before providing information about agents orexecuting any other requested command. In a subscribing entity role,controllers must be able to authenticate other controllers whenoperating in a cluster.

In a subscribing entity role, an agent must be able to authenticatecontrollers and agent discovery services. Controllers are authenticatedbefore an agent will accept commands to acquire data, etc. Agentdiscovery services are authenticated before the agent will disclose anyof its network configuration settings.

In a subscribing entity role, a console authenticates controllers beforepermitting a user to enter their credentials (such as a username andpassword).

In a trust domain entity role, an agent discovery service willauthenticate itself to agents before agents transmit their networkconfiguration information. In a subscribing entity role, an agentdiscovery service must authenticate controllers before accepting anycommands, such as disclosing the network configuration information foran agent.

In a trust domain entity role, the end user must authenticate himself orherself to a controller before they can be provided access to thatcontroller or controller cluster. In a subscribing entity role, usersmust authenticate the controller they connect to before they input theircredentials to gain access.

In order to create and maintain the trust domain, the following fouroperations must be implemented and supported in the invention: (1)creation of a certificate authority, (2) signing of certificates fortrust domain entities, (3) maintenance of a revocation list for trustdomain entities, and (4) authentication and revocation checkingmechanisms in a subscribing entities. Each of these steps will bediscussed in detail.

In the first step, a certificate authority must be created to anchor thetrust domain. The role of the certificate authority is to issueidentities and credentials to trust domain entities by signingcertificates for those trust domain entities. In one embodiment of theinvention, the X.509 standard is used for certificates, which specifiesthe format, signing algorithms and standards, and methods for validationof a certificate. The certificate authority is a component of onecontroller within a trust domain. If there are multiple controllerswithin the same trust domain, one is designated the master and becomesthe certificate authority for the entire trust domain. The newcertificate authority is called the trust domain certificate authority(TDCA), and is embodied by a public/secret key pair, as well as acertificate representing the certificate authority. The TDCA certificatecan be used by any subscribing entity to verify whether a certificatepresented by a trust domain entity is valid; therefore, all subscribingentities must receive a copy of the TDCA certificate before they canbegin to operate with components that are in the trust domain.

The second step is the signing of certificates for trust domainentities. The primary operation of the TDCA is to sign certificates forTDEs, thereby making them members of the trust domain. This isaccomplished according to the appropriate standards associated with useof X509 certificates within a functioning PKI. In short, an entity thatwants to become a TDE (for example a new controller) must generate apublic/secret key pair along with a certificate signing request (CSR)and submit the CSR to the TDCA. This submission may either be over anetwork connection or out-of-band (e.g. manually input by a user intothe TDCA). The TDCA must validate whether the CSR is valid (in oneembodiment of the invention, this is done manually by the user), andthen the CSR must be digitally signed by the TDCA, using the secret keycorresponding to the TDCA's public key (which is embedded in the TDCACertificate). Signing the CSR results in creation of a certificate; thisis then passed from the TDCA back to the requesting entity either vianetwork connection or out of band. Once the newly created certificate isinstalled in the requesting entity, it becomes a TDE and may now operatewithin the Trust Domain.

FIG. 5 depicts the creation and deployment of a TDCA. FIG. 5 depicts amethod for certificate authority creation 502 and certificate signing504. A User 500 requests initialization through a web-based userinterface in a step 515. The Master Controller 505 bootstraps thesigning keypair and master certificate in a step 520. The User 500 isprompted for and sets passphrase to protect the certificate authoritysecret key, the certificate authority expiration date, and defaultexpiry for subordinate certificates in a step 525. The certificateauthority secret key is stored on the master controller disk andprotected with a user-set passphrase in a step 530. A certificaterevocation list is created, signed, and published in a step 530. Themaster certificate authority embeds the public key and currentcertificate revocation list into Agent installation package in a step535. A User 500 requests that a Controller generate a certificatesigning request in a step 540. The Controller 510 creates a keypair anda certificate signing request for use in authentication to Agents 500 ina step 545. The User 500 receives the certificate signing request in atext format from the web-based user interface in a step 550. TheController secret key is stored on the Controller 510 disk and protectedby an encryption method that does not require user input for operationin a step 555. The User 500 requests signing from the certificateauthority on Master Controller 505 by supplying the certificate signingrequest into the user interface for the Master Controller 505 in a step560. The Master Controller 505 requests a passphrase to unlock acertificate authority secret key and authorize signing check mastercertificate authority expiry in a step 565. The User 500 inputs thecertificate authority secret key passphrase in a step 570. The MasterController 505 signs a certificate signing request and returns thesigned certificate to a user in the web-based user interface in a step575. The User 500 receives the signed certificate from the MasterController 505 and supplies it to the target Controller 510 through theweb-based user interface in a step 580. The Controller 510 accepts thesigned certificate and validates that it matches secret key andcertificate signing request previously created in a step 585. TheController 510 stores and uses the certificate for connecting to Agents500 in a step 590.

The third step is the maintenance of a revocation list for trust domainentities. The TDCA can issue certificates to entities to make them partof the trust domain. However, it must also be able to revoke thosecertificates, allowing subscribing entities to identify when a given TDEshould no longer be trusted. For example, if a user had two controllerswithin their trust domain and subsequently decided to remove one (e.g.,sending it away for maintenance or decommissioning it because the extracapacity was no longer required), a method must be in place to ensurethat controller's certificate is no longer trusted by the trust domain.In one embodiment of the invention, this is accomplished by publicationof a certificate revocation list (CRL). The CRL is created and signed bythe TDCA when a user wants to remove an entity from the trust domain.The CRL contains a list of all certificates previously issued by theTDCA that are no longer valid. Subscribing entities can obtain a copy ofthis list and use it in conjunction with the TDCA certificate tovalidate whether a TDE has a valid, non-revoked certificate when itcommunicates with an SE.

FIG. 6 depicts one embodiment of a method of certificate revocation listmanagement according to the present invention. A User 600 requestsrevocation of a specific controller through the web-based user interfacein a step 612. The user is prompted by the Master Controller 602 for apassphrase to a certificate authority secret key in a step 614. The userenters the certificate authority secret key passphrase in a step 616 andthe certificate authority updates the certificate revocation list withthe certificate from the selected controller and signs it using acertificate authority secret key in a step 620. The User 600 thenselects an update method in a step 622 and the Master Controller 602creates jobs for all subordinate controllers to update the certificaterevocation list based on the method selected by the user in a step 624.The agent discovery service is updated with new certificate revocationlist and agents contacting the agent discovery service download andauthenticate the certificate revocation list in a step 626. Thecontroller may also contact an agent and instruct it to install the newcertificate revocation list. All controllers launch a job on all agentsto attempt connections to any agents and update the certificaterevocation list on all agents in a step 628. The Agent Discovery Service604 authenticates a connection with the Master Controller 602, validatesthe signatures on the certificate revocation list and provides it toagents as they periodically contact the Agent Discovery Service 604 in astep 630. The Agent Discovery Service 604 authenticates to agents suchas Agent 606 if an updated certificate revocation list is available andprovides the list to the Agent 606 in a step 632. The Agent 606occasionally polls the Agent Discovery Service 604 and updates the AgentDiscovery Service 604 with network settings in a step 634. If an updatedcertificate revocation list is available at the Agent Discovery Service604, the Agent 606 will download the certificate revocation list, verifythe signature on the list, and install the list in a step 636.

The Controller 608 looks up records for all agents and attemptsconnections to the agents for the purposes of updating the certificaterevocation list on the agents in a step 638. The Controller probes allagents for the purposes of updating the certificate revocation list andupdated the certificate revocation list on agents that were notpreviously known in a step 640. The Agent 610 authenticates andauthorizes the connection with the Controller 608 in a step 642. TheAgent validates the signature on the certificate revocation list againstthe certificate stored in a secure storage in a step 644. If thecertificate is valid, the certificate revocation list is updated in astep 646 and used for subsequent transactions.

FIG. 7 is a flow diagram depicting the steps involved in the use of aCRL. A Controller 700 obtains network information for a desired Agent710 from an agent discovery service and initiates a network connectionin a step 720. The Agent 710 accepts the network connection and obtainsthe Controller's certificate in a step 730. The Agent 710 provides itsown certificate back to the Controller 700. The Controller 700 validatesthe Agent certificate and if the certificate is for an existing Agent710, the Controller ensures that it matches the certificate on recordfor that Agent in a step 740. If the agent has never been seen before,the Controller creates an Agent record and stores the Agent certificate.The Agent 710 validates that the signature on the Controller certificatewas performed by the trust domain certificate authority the Agent trustsin a step 750. The Agent 710 verifies that the Controller certificate isnot contained in the trust domain certificate authority certificaterevocation list in a step 760. The Agent fully accepts the Controller700 connection and prepares to process commands issued by the Controller700 in a step 770. The Controller 700 sends a job to the Agent forexecution and collects the results in a step 780.

The fourth step if the implementation of authentication and revocationchecking mechanisms in all subscribing entities. Subscribing entities(SE) must be able to authenticate members of a trust domain in order tointeract with it. For example, agents need to be able to authenticate acontroller before accepting commands from it to acquire data; users needto be able to authenticate a controller before typing their username andpassword into the console to gain access. A subscribing entityauthenticates a TDE by validating that the TDE's certificate was issuedby the TDCA. To do this the subscribing entity must have a copy of theTDCA certificate. In one embodiment of the invention, the TDCAcertificate is part of the installation package for the agent andconsole software. When a TDE connects to an subscribing entity (or viceversa), the TDE provides a copy of its certificate to the subscribingentity and performs a challenge operation to validate that it also hasthe secret key that corresponds to the public key embedded in thatcertificate (in one implementation this is done using the Secure SocketsLayer protocol). The subscribing entity then validates the TDEcertificate by verifying the digital signature embedded inside of it. Ituses its copy of the TDCA certificate to perform this operation. It thenchecks the TDE's certificate against the CRL. If the TDE certificate isnot on the CRL, the connection is then authenticated and the subscribingentity can continue its communication with the TDE. FIG. 7 is a flowdiagram reference implementation of authentication with revocationchecking between a TDE (controller) and a subscribing entity (agent).

In addition to the maintenance of the trust domain, there must also be amethod to uniquely identify certain subscribing entities that do notfunction as a trust domain entity—that is to say, there needs to be amechanism to have a unique, authenticated identifier for an entity wherethat identifier was not issued by the TDCA. In particular, there must bea method for agents to create or obtain a unique identity so they canparticipate in operations within a given trust domain. This is requiredin order to ensure that the results of any data acquisition can be boundto a single, unique agent so a user knows a given set of acquired datacame from a given agent—identifying the source of information in eitheran IR or EED event is critical.

One embodiment of the invention uses the following process to provide amethod for agents to create a unique credential and subsequently utilizethat credential within a trust domain.

Upon installation, the agent must have the TDCA certificate installedwith the agent software to ensure the agent can authenticate TDEs. Theagent is also configured with the network information necessary tocontact the Agent Discovery Service for the Trust Domain.

At first startup, the agent creates a self-signed certificate; that is,a secret/public key pair with a corresponding certificate thatencompasses the public key, whereby the certificate was digitally signedby its own private key. This particular embodiment uses the X509certificate standard for specifying format and signing algorithms. Thepublic/secret key pair may be, for example, an RSA key 2048 bits orgreater in length, generated through use of software adhering to PublicKey Cryptography Standard #1. However, any algorithm and key lengthwhich adhere to public key infrastructure standards may be used.Software is used for generating random data during the key generationprocess (an important aspect of ensuring a key is reasonably secure andunique).

Two possible methods are now used to “enroll” the agent. Enrollment isthe process of registering an agent's existence with the controller andrecording its certificate. An agent may be enrolled either via the AgentDiscovery Service or through a direct connection from a controller tothe agent using one of the following steps.

An agent may enroll with a controller using a direct connection. When acontroller connects to an agent, it transmits its own certificate andreceives a copy of the agent's self signed certificate. Thesecertificates are used to establish an SSL connection, which is then usedfor all subsequent communication between controller and agent for thatsession.

The controller looks up the agent certificate to see if an agent recordexists for it. If it does not, it creates a record for the agent andstores a copy of the agent's certificate inside of it. Any dataretrieved from the agent (e.g. a process listing, files from the harddrive, etc) is associated with the newly created agent record. Thisprocess is referred to as enrollment.

If the controller has seen the agent certificate before, it identifiesthe agent record for that certificate—in other words, the agent hasalready enrolled. Any data retrieved from the agent is associated withthis pre-existing agent record. See FIG. 7 for an illustration of thisprocess.

Alternatively, an agent may enroll using an agent discovery service. Ifthe agent is configured to contact an agent discovery service, the agentinitiates a connection and authenticates the agent discovery serviceaccording to its certificate by using its local copy of the TDCAcertificate and TDCA CRL. See FIG. 7 for an example of howauthentication of TDEs, such as the ADS, functions. The ADS records theagent's certificate and network configuration settings which aretransmitted by the invention to the ADS. See FIG. 4 for moreinformation. When ADS is polled by a controller for a list of newagents, the record for the agent, along with its certificate, isprovided to the controller.

The controller looks up the agent certificate to see if an agent recordexists for it. If it does not, it creates a record for the agent andstores a copy of the agent's certificate inside of it. Any dataretrieved from the agent (e.g. a process listing, files from the harddrive, etc) is associated with the newly created agent record. Thisprocess is referred to as enrollment.

If the controller has seen the agent certificate before, it identifiesthe agent record for that certificate—in other words, the agent hasalready enrolled. Any data retrieved from the agent is associated withthis pre-existing agent record. See FIG. 4 for an illustration.

Once all three classes components of the system are deployed (console,agent, and controller), the security infrastructure established, andagents are registered and reachable by the controller (if operating innetworked mode), collection and analysis functionality can be used.

In one embodiment of the invention, users interact with the console toidentify a series of collection and analysis commands to be executed.Such a series of commands is referred to in the context of the inventionas a script. The user also specifies a set of inputs for the script anda time or times for the script to be executed. A script, when combinedwith a set of inputs to operate on (e.g. a list of agents to run thescript on, a set of data to analyze, etc) and a time or times to execute(e.g. “now”, “Aug. 29, 2007 at 11:59AM ED”), is called a job. In oneembodiment, commands are in a pipeline; that is, a sequential executionof one command to the next. A critical failure of any step within theseries causes the entire job to quit. Other implementations couldinclude the ability to operate in a grid or tree structure, where theoperation of commands could depend on the state and results of aprevious command. Failure of any step would not necessarily mean thatthe job would halt if subsequent processing directives were provided toindicate execution could continue. Such an approach would require thestructure for a script to be very similar to a computer programminglanguage.

As inferred above, multiple commands may be “chained” together; acommand to collect a certain series of data could be followed by acommand to analyze that data. The outputs of one command must match therequired inputs for another command in order for them to be chained.

The controller is responsible for executing the job. It reviews thecommands to be executed along with the inputs for each of thosecommands. Commands are either directives to marshal to a set of agentsor a series of analyzers via the analysis service (see FIG. 3). Thecontroller may also implement parallel execution of commands within thejob's script if the individual commands are found to be fullyindependent of one another (note this is true whether the script is ofpipeline form, or grid/tree). In embodiments with more than onecontroller (e.g., configured as a cluster as described herein) commandsmay also be sent out to different controllers for execution.

When a command is executed, it returns a set of data that may containthe intended set of results (e.g., a set of data from an agent, theresults of an analysis), one or more error documents that identify why acommand failed to execute, or both if a command was partiallysuccessful. These results are referred to as an audit or audit result.The controller stores this information and performs various operationson it to make it available for use by the end user, including indexing(making the data available through a search engine). The data can thenbe accessed by the user for viewing, used as input for an analysis,marked up by the user during the review process, and referenced in thecreation of reports or other documents within the invention.

FIG. 8 provides a flowchart outlining a job execution process accordingto the present invention. The user defines a series of commandsincluding order of execution and conditions in a step 802 to create ascript 804. The user specifies inputs to the script as well as time ofexecution of the script in a step 806 creating a job 808. The job isvalidated by the invention including commands in the job's script,inputs, and time of execution in a step 810. The job is examined todetermine whether the job is correct in a step 812. If the job is notcorrect, errors are reported to the user in a step 814 and the user isallowed to redefine the script. If the job is correct, the controlleridentifies commands in the job that can be executed in parallel or inseries and begins execution of the tasks in a step 816. The commands areexamined to determine whether they are in parallel or in series in astep 818. If the next command is series 820, the next in-series commandis executed and the results are recorded in a step 822. The executedcommand is examined to determine whether it is the last command in astep 824. If it is not the last command, the next command is examined todetermine whether it is in parallel or in series in step 818. If thelast command has been executed, the system performs post collectionactions such as storing data and indexing data in a step 830. If theexamined command to be executed is in parallel 826, the system executesall parallel commands on available controllers and records results in astep 828.

Commands for collection of data are sets of instructions to bedispatched to one or more agents. The instructions have a series ofparameters defined that identify the data the agent is to collect. Ifworker modules were also implemented, these commands would includeinstructions on how to modify the computer system the agent was runningon. Parameters identify characteristics of data on the target systemthat should be collected. This allows a user to specify only what theywant to retrieve, reducing the amount of information transported back tothe controller from the agent. This “precision strike” collection methodassists in offsetting the challenges posed by ever-increasing storagemedia sizes on computer systems. Extensive specification of parametersis only implemented for those classes of data that are large orcumbersome to manage; for smaller, easier to collect data simply“getting everything” and allowing the user to filter it afteracquisition is more efficient.

Several classes of filtering are defined to assist users in narrowingthe scope of their audit depending on the type of data they arecollecting. In general, there are two primary methods for filtering:filtering based on the metadata for an item, and filtering on contentcontained in the item itself. In one embodiment, all of the dataspecified comes from systems using the Microsoft Windows operatingsystem. The data being referenced is representative of most commoncomputer systems; other data could be added depending on the goal of theinvestigator, the focus of a specific invention implementation, or thespecific computer system or device the data is being gathered from.

The type of data acquired will vary by embodiment and need of the user.Various parameters may be used for filtering, which may be based onmetadata or content. File listings and file contents for both regularand deleted files may be filtered using metadata by file name, pathname, creation date, modification date, access date, file size, filetype, or other information depending on available metadata. The filelistings and file contents may also be filtered by content or file hash.The entire contents of a computer system hard drive or other storagedevices may be filtered by disk size, disk volume, or logical driveletter for each device. Portions of or the entire contents of memory ona computer system may be filtered by memory location using the metadataof the memory. It may also be filtered by contents or owning processusing the content of the data.

Metadata or content items may be compared to user-supplied arguments.Those arguments typically take the form of a value to compare againstand an operator (e.g., =, >, <, >=). In one embodiment, comparisons mayalso support the use of regular expressions by users to specify apattern for matching against metadata or content. Regular expressionsare strings of characters or byte sequences that are used to describe ormatch a set of strings or byte sequences according to certain syntaxrules.

Once data is collected by the agent, it is retrieved by the controller,potentially stored in a forensics file format if one is not alreadypresent, stored and referenced in the system, indexed for search, andmade available for access by end users. The following sections detailthis process. An overview of the data collection process is presented inFIG. 9. A user initiates a job for data collection from the console 900to the controller 904 in a step 902. The controller 904, using an agentdiscovery service or direct connections, contact agents and transmitsjob information, including script of commands to run in a step 906. Theagent 908 collects data per parameters specified in the job andorganizes the data into XML documents containing the results in a step910. Optionally, the agent may wrap results in a forensics file format912 container before return to the controller 904 in a step 914. Thecontroller 904 retrieves results from the agent and prepares them forstorage in a step 916. Results from the agent are packaged together intoan audit result 918 in a step 920. If individual result documents werenot already packaged in a forensics file format container, thecontroller 904 places them in such a container. Results are assignedrequired metadata so they can be accessed by a user and understood bythe system. Results are then indexed for use by the search engine andmade available for viewing and analysis in a step 922.

For both security evaluation and electronic discovery events, aninvestigator may require assurances that acquired data has not changedsince the time it was collected. The most common method of providingthis for evidence (both digital and physical) is through use oftamper-evident processes. A tamper-evident process may not directlyprevent modification of evidence or data, but it makes highly improbablethat the modification can take place without that modification beingdetectable.

Through use of various cryptographic techniques, it is possible topackage digital information into formats that are tamper-evident. Oncepackaged, if data within the package is modified, the modification isdetectable. In one embodiment, an open format is used for providing asubset of this capability. The Advanced Forensics Format (AFF) providesmethods for storing data in a container that, in turn, containsinformation about those contents sufficient to detect if they wereaccidentally modified. The AFF container contains secure hash checksumsof the information contained inside of it. By examining those checksumsand comparing them to the data inside of the AFF container, it ispossible to detect if information was accidentally modified.

The scenarios outlined herein assume a configuration of the system wherethe controller is connecting to the agent over some form of network.However, many of the analysis, markup, and reporting capabilities of thesystem are still usable if data is imported directly into thecontroller. In one embodiment, the system includes a mode of operationfor the invention where it can operate out-of-band (OOB): the agentexecutes locally on a computer system and saves the acquired data tosome form of removable media (e.g. a USB memory storage key, externallyconnected hard drive, floppy disk). The removable media is then takeneither to a workstation where the console is installed or directly tothe controller itself and then imported into the system. This providesseveral advantages. Firstly, operating OOB is sometimes more efficientfor large collections if networks are slow or congested. In a scenariowhere a user is trying to collect significant amounts of data (theentire contents of a hard drive, perhaps a large set of files), using anOOB method to directly interface with the target system to acquire datamay result in a faster collection. Secondly, some target systems are oncompartmentalized networks or not networked. In these scenarios an OOBmethod for collection provides a work-around for performing a collectionwhen a controller cannot connect directly to the agent over a network.Thirdly, an OOB use of the agent may have a reduced footprint on thetarget system. In some circumstances a user may wish to minimize theimpact on a target system. For example, if the user were conducting aninvestigation against a fellow employee and wanted to collect data fromtheir computer system, the user might want to avoid persistentlyinstalling the agent to minimize the risk of detection. By using theagent in an OOB configuration, the user could place it on removablemedia (e.g., a USB key), go to the target system, run the agent, andsave the data to the same removable media.

Data must be normalized and stored in an efficient manner in order forthe invention to utilize it for analysis, search, and collaborativereview. Much of the data described herein can be represented in rowswith each row containing several fields, the combination of whichdescribes a data item. Each class of such data must be described in theform of a schema—a formal description of the data using a commonsyntax—in order to provide the necessary structure for performingrelational comparisons between data sets. For example, in one embodimentof the invention a process listing contains multiple rows of processinformation, with each row containing a process name, process identifier(a number assigned by the computer operating system the process isrunning on), the full path and filename to the executable fileassociated with the process, arguments (e.g. options supplied atrun-time), start date/time, and the length of time the process has beenrunning. A file listing contains, among other things, the full path andname of the file, the created, last modified and last accessed time, andfile size (in bytes). The date/time fields and path/filename fields mustadhere to the same syntax in order for relational operations to bepossible across them (e.g. “show the creation date for all of the filesassociated with currently running processes”).

In one embodiment of the invention, the requirement for formal schemaand storage syntax is met by using Extensible Markup Language (XML). XMLis a general purpose markup language (a standard syntax for text andinformation about that text) that facilitates the exchange of structuredinformation. The various XML standards provide both a method fordescribing the structure of data in a schema and a syntax for storingthe data itself. The invention defines a set of schemata for the varioustypes of data collected from a target system through the agent. Allcomponents of the invention adhere to those schemata when operating onthat data: the agent formats the acquired data into the proper schema,the controller ensures it does not modify that data once it is acquired,and the console understands how to parse and display the data adheringto the various schemata. FIG. 10 illustrates how one component of thesystem changes representation of data from non-XML into XML formats.System specific data structures 1010 are translated by the Agent in astep 1020 to create an XML representation of the same data 1030. Oneskilled in the art would recognize that a plurality of differentlanguages may be used to normalize data in the system.

In addition to formally describing the structure of individual dataitems, the system preferably defines higher order structures in which toorganize individual data items and groups of data items in order to makethe overall data model comprehensible to the end user. The following isa list of data objects, the contents of each data object, and thefunction of the data object.

Row Item. The row item contains collected data that is row/fieldoriented. Direct storage obtained from a target system can berepresented as a row-oriented, field-based entry. Examples includeprocess list items and file list items.

Content. Content contains multiple row items that share a common schema,or contains verbatim information from a target system that should not betranslated into a document of row items according to a schema. Row itemcontent examples include: complete process listing and complete filelisting. Verbatim examples include entire files from the file system ofthe target system, hard drive contents from the target system, andsystem memory contents from the target system

Entity. Entity data objects are generic objects that contain metadataand content. Entities are a generic form of object within the system.Entities directly store metadata and content, or have reference directlyto content. Entities may also contain collections, described below.

Collection. A collection is a generic object that contains a list ofother objects. Collections are used to represent a list of otherobjects, which are typically entities.

Document. A document contains content and metadata about the content.Documents wrap contents and include additional attributes andinformation about those contents that are necessary for the system tofunction.

Result. A result stores the results of an audit from an agent or ananalysis in the form of a collection of documents. A result provides aset of metadata about the audit or analysis including information aboutthe script used to run the audit or analysis.

Result set. A result set contains a collection of results for a givenjob and associated metadata. Result sets contain metadata about the jobitself (such as the script used for the job) along with a collection ofresults.

Markup. A markup contains information about attributes as applied toindividual row items within a document. Markups are used to store theinformation necessary for row item attribution, which is the methodprovided by one embodiment of the present invention for allowing usersto mark or annotate individual row items within a record-orienteddocument. This permits use cases such as setting visible flags or tagson data items, or enabling workflow at a level more granular than acomplete document.

Attribute. An attribute contains a name, content, and list of objects.Attributes provide a mechanism for users to “mark” data within thesystem. The attribute object stores the information necessary toidentify the attribute and all of the objects it is applied to withinthe system. Some implementations of the invention may include row itemswithin this context, or may use special characteristics of the markupobject discussed above to address row-level attribution.

Search Result. A search result contains a search query and evaluation ofthe query returns a list of the entities responsive to the query. Searchresults store search queries specified by the user or other componentsof the system. The function of the search result is to create a dynamicgrouping of documents that contain the terms indicated in the query.Each time the query is evaluated (i.e., each time the search result isviewed) a document is returned containing references to entities withinthe system that contained the search terms specified in the query.

Library. Libraries organize all entities of a given type into a list.For example, the list of all audit and analysis results is in the auditresult library and a list of all documents is in the document library.

Workspace. A workspace is a set of libraries containing all objectswithin an access domain in the system. Workspaces are the “top level”entities in the system. They contain a series of libraries that in turncontain all the data for a given access domain. In this context, anaccess domain is a set of data that share access control rules.

Each object type is represented as an XML document, ensuring consistentsyntax across the system for data representation. When an object isstored, a way to refer to and access that data is also provided so thatsubsystems within the system and end users can read the data andinteract with it. In one embodiment of the invention, interaction isfacilitated through use of Representational State Transfer (REST). RESTis a style of software architecture that outlines how resources aredefined and addressed. Expressed simply, REST has requirements forproper functionality on a system.

The system is preferably composed of client components and servercomponents, where the client is concerned primarily with user interface,and the server manages data storage and access. The system provides thisthrough the components of the console (client) and controller (server).Additionally, the controller acts as a client when interacting with theagent, which in turn acts as a server during those transactions.

The system is preferably also stateless, in that every request fromclient to server preferably contains all the information necessary toprocess that request and not rely on subsequent or previoustransactions. Any state maintained within the system is maintained bythe client. The invention is stateless in all transactions betweenconsole, controller, and agent—each request is independent and containsall the information necessary to execute it, whether between console andcontroller or controller and agent.

The system preferably functions with a cache. Responses from server toclient are preferably explicitly marked as cacheable or non-cacheable.Cacheable means that a response from the server can be later re-used bythe client for future equivalent requests. The system provides for thisbetween console and controller.

The interface between all components of the system is preferably thesame or similar. REST is defined by four interface constraints:identification of resources, manipulation of resources throughrepresentations, self-descriptive messages, and hypermedia as the engineof application state. Resources are “any information that can be named”and therefore accurately describes all of the data objects describedabove. Within REST, and within one embodiment of the invention, theidentifiers for resources are Uniform Resource Identifiers (URI) asspecified by Internet Engineering Task Force (IETF) Request for Comments(RFC) 3986. A representation is “a sequence of bytes, plusrepresentation metadata to describe those bytes.” In essence an entityas defined above is a representation in this context. More specifically,XML documents are the model for all data within one embodiment of theinvention. A piece of data may be changed by taking a copy of that data,changing it to the desired state, and then sending that copy to thecomponent within the invention responsible for processing that data.Messages are self-descriptive. This captures the concept ofstatelessness and “cacheability” as described above; all messages sentfrom client to server contain everything inside of them that isnecessary to understand them, while messages from server to client areexplicit in stating their cacheable status. Hypermedia as the Engine ofApplication State: Given that all resources are manipulated bytransferring representations of those resources between components ofthe system, and the system fully operates on self-descriptive messages,it is clear that the combination of these two concepts make up theengine by which an application may derive state. In a REST-compliantinterface, the responsibility for state maintenance is fully on theclient, ensuring server components do not need to comprehend it, andmessages between components do not have special methods for transactingit.

A system according to the present invention provides for each of theseby assigning URIs to every data object within the system, representingthose objects as XML documents, and utilizing the Hypertext TransferProtocol (HTTP) over SSL for network transactions between the console,controller, and agent.

A system according to the present invention is preferably a layeredsystem, meaning it is composed of hierarchical layers where each layermay not “see” beyond another layer with which it is interfacing. Thistends to reduce complexity of the design. A particular system may haveany number of layers depending on the complexity of the system.

A system according to the present invention extends client functionalityby allowing clients to download and execute code from the server,typically in the form of scripts. It is typically optional in a RESTsystem. In one embodiment of the invention, this is used when providingenhanced capabilities for user-generated reporting.

Each data object within the invention is represented by an XML document.Containers are represented by a special form of XML document—asyndication feed. Syndication feeds provide a summary of availablecontent, typically in the context of a web site. Using this structure torepresent containers as described above provides a consistency with therequirements for REST as stated above. There are a number of standardsfor formatting and providing syndication feeds within a softwarearchitecture. One embodiment of the invention uses an Atom syndicationfeed, which fully adheres to REST software methods.

FIGS. 11-14 illustrate REST in the context of initiating a job from theconsole to collect various data from an agent. FIG. 11 illustrates a jobwith accompanying script in XML. FIG. 12 demonstrates how one embodimentof the invention works using REST in a sample transaction. FIG. 13illustrates how status updates regarding ongoing operations on thecontroller can be retrieved by a console using Atom syndication feeds ina REST compliant fashion. FIG. 14 illustrates the REST model in anexchange between controller and agent.

The process depicted in FIG. 12 begins when a console 1200 creates anXML document representing a job as depicted in 1202. The consoleincludes the script to be run and the identities of hosts to execute thejob against. The job is submitted to a defined location for job creationon the controller 1204 in a step 1206. The controller 1204 takes thedocument from the console 1200, validates it, and adds additionalinformation such as the newly created document's uniform resourcelocator plus the uniform resource locators of related resources thatmust now accompany the job such as a container for results, attributes,logs, and audit trails in a step 1208. The document is also returned tothe controller 1204 in a step 1210. The job document with informationadded by the controller 1204 is depicted in box 1212. To run the job,the console 1200 obtains a fresh copy of the job document and modifiesthe contents of the job document indicate that it should be run in astep 1214 as depicted in the updated job document 1216. The job documentis then submitted back to the controller. The controller 1204 returnsthe document to the console 1200 with a success code indicating that thejob was accepted for processing in a step 1218. An excerpt of the jobdocument is depicted in box 1220.

The process depicted in FIG. 13 begins when a console occasionally pollsa controller 1302 to access the feed of all available updates in a step1304. An example of an http get command to obtain such an update isdepicted in 1306. The controller 1302 then returns a document containingthe requested set of update information in a step 1308. The console 1300may use this information to update the user interface or perform otherstate management operations. An example of a document containing updateinformation is depicted in 1310.

The process depicted in FIG. 14 begins with a controller 1400 contactingan agent 1402 and transmitting a script document after receiving a jobfrom a console along with execution instructions in a step 1404. Oneexample of a script document is depicted in 1406. The agent 1402 thenaccepts the document from the controller 1400, validates it, and createsa results document with the uniform resource locators the controller1400 should use to retrieve results in a step 1408. One example of auniform resource locator the controller 1400 should use to retrieveresults is depicted in 1410. The controller 1400 uses this uniformresource locator to poll the agent 1402 in a step 1412. The agent 1402responds with a message and the results if they are ready or an errormessage suggesting that the controller 1400 request results again laterin a step 1414. An example of a response message is depicted in 1416.

Data stored within the system fits into two broad categories: metadataand content. Metadata is “data about data.” Metadata includesinformation about a data object that is required to describe it to therest of the system or an end user. Typical metadata fields within theinvention include, but are not limited to: identity, name, date and timeinformation, and ownership and security information. Within a REST basedsystem the identity of a data object is the URI to that resource, e.g.,https://mandiant/workspaces/1/documents/all/99. Identities are uniquewithin the system. Each identity describes one and only one data object,and each data object has one and only one Identity. A name for an objectmay be defined by users and are typically included for end userpurposes. Note that a name is not an identity. Much like human names, itmay not be unique—several objects could be named the same thing, whileeach would have its own unique, individually referenced, identity. Dateand time information includes creation date/time, modificationdate/time, and any other temporal data about (as opposed to within) anobject. Ownership and security information includes definitions forwhich user “owns” an object, and which users may access, modify, ordelete an object.

Content is the data within the data object in question. For example, thecontent of File Listing Audit, it would contain a list of files. For afile acquired from a target system's hard drive, it would be the contentof that file (a word processing document, an executable program, anemail, etc).

Data is stored within the system using a combination of “document on afile system” and “data within a relational database” techniques,depending on the functionality requirements for the data. Note, however,that this is not a strict requirement of the present invention. Themethods described may be used in some embodiments. Other mechanisms thatsupport an implementation of a REST-based software architecture may beutilized in other embodiments. For example, all information could bestored within a relational or object-oriented database. The usage andperformance requirements drive the selection of the specific datastorage and management architecture.

In one embodiment of the present invention, most information is storedwithin a file on the controller, with identity and reference informationstored within a relational database to support rapid lookup. Bothcontent and metadata are kept within XML files on disk, whilerelationships between objects (e.g. “the list of objects an attribute isapplied to”) are stored within a relational database.

FIG. 15 illustrates this process. A new input data document 1500 issubmitted to the data intake process 1502 in which an identity for a newobject is created. The file is stored on a disk within a file systemrepository 1504 in a step 1506. The identity of the new object, path tothe object on disk, and relationship information within the object arestored within a relational database in a database management system 1508in a step 1510. The object's metadata and contents are indexed by thesearch system in preparation for a search in a step 1512.

As previously mentioned, in one embodiment of the invention, mostcontent is stored on the file system of the controller. However, thereare many situations when the console may need to access portions of theinformation contained in a data object as opposed to the entire objectitself. This is most common when a data object is large, containingeither a significant amount of data or in the case of record orientedcontent, a large number of rows. In these cases retrieving the entireobject at once may create performance issues, bottlenecking controllerperformance, computer network performance, or console performance as itattempts to organize data to render it visually for an end user. Toovercome this problem the controller supports the concept ofvirtualization of data.

Virtualization is the process of organizing a data object such thatportions of it may be returned to a requester based on a set of definedoffsets. In the case of record-oriented data, the offsets might bespecified as row numbers. In the case of non-row oriented documents(e.g., a binary file from a computer system), the offsets might bespecified as byte-offsets within the document.

One embodiment of the invention provides this method through the use ofa relational database. When a data object is requested by a console, thecontroller “virtualizes” it—that is, loads it into a relationaldatabase, automatically organizing it into tables and fields based onthe data object's structure. The console can then query the relationaldatabase containing the data object, requesting subsets according to itsrequirements. For example, given a File Listing Audit containing 500,000rows, the console might request several hundred rows at a time; giventhe constraints of user interface devices such as monitors, the usercould only visualize a small set of rows at any given time. The consolerequests only the relevant set of rows that the user wishes to view,preventing the need to transfer large amounts of information betweencontroller and console before those rows can be visualized.

If a data object has a well-defined structure (for example, if it iscontained in a well-formed XML document), the controller has the abilityto automatically load the document into a relational data table. If thedata object follows a well defined schema that the controller has accessto, more advanced query features can be made available to requestingentities—for example, the ability to sort according to data type rules(e.g., sort numerical field in ascending order) can be made available ifthe controller has explicit information about the schema the data objectfollows. If a schema is unavailable, the controller can “guess”,extracting records from the XML document and placing them into fieldsthat are compatible with, but perhaps not optimal for, the individualfields contained in each record.

Search engines provide capabilities for retrieving information stored ona computer system according to a series of search terms and modifierscollectively referred to as a search query. The most common form ofsearch engines are those applied to searching information on the WorldWide Web. Search engines are also commonly employed in various forms ofsoftware, from desktop applications to enterprise infrastructuresystems.

The present invention provides search engine capabilities for findinginformation across any set of acquired or user-created informationwithin the system. The search engine is embedded in the controller, andis tightly integrated with data storage services. Given the common usecases for the invention, the data sets stored and searched within thesystem could be quite large (on the order of terabytes of information).Given that, various data organization and management methods areemployed within the system to balance performance and resourceutilization within the controller environment.

One embodiment of the invention utilizes the Lucene search engine fromthe Apache Foundation; however, any search engine may potentially beadapted for use within the system. The description of the dataorganization and management methods will be in the context of Lucenedata structures, but the generic concepts are potentially applicableacross any search engine utilizing modern search methods.

The search system within the present invention is made up of fourprimary classes of components: search documents, indexes, stores, andsearch hives.

Search documents differ significantly from typical data object documentswithin the system. A search document is created, typically in memory,when the search engine begins to index a document. It is a container ofinformation about the data object being indexed, and may be used tocontain special information depending on the focus of the search indexbeing created. It also contains all of the content of the original dataobject document being indexed. One or more search documents are createdfor each input document indexed by the search engine.

A search engine index is a set of data that is constructed from a corpusof information; its contents are organized to optimize the speed andperformance of finding the subset of relevant documents within thecorpus that match a search query. In order to add a new document to asearch engine it is “indexed.” Indexing is the process of analyzing adocument and producing a set of information that can be used to respondto search queries. When a search query is submitted to a search engine,this subset serves as a substitute for scanning the entire originaldocument. The information produced during indexing is merged into asearch engine's index in order for it to be included in the scope ofsearch queries sent to the search engine. Search engines may have one ormany indexes that are reviewed when analyzing a search query. Indexesmay be optimized for different forms of searches, depending on the goalsof the system.

In the context of the invention's search system, a store is a group ofindexes that are optimized for the same class of search queries. Oneembodiment of the invention utilizes two stores within the system: ametadata store and a content store. The metadata store contains indexesof metadata from all data objects within the system. The content storecontains indexes of content from all data objects within the system. Ifa cluster of controllers are operating together, there will be onemetadata store on the master controller for the cluster and multiplecontent stores (typically one per controller in the cluster).

A search hive is the collection of all stores within the search system.One embodiment of the invention utilizes a single search hive within anycluster of related controllers. That is to say, for any set ofcontrollers in an embodiment of the invention, there will only be oneactive search hive. All searches requested by any component of thesystem or an end user are serviced by this single hive. It is made up ofthe entire set of search stores contained on all controllers in thecluster.

FIG. 16 depicts the search system component classes and theirrelationships. An input document 1600 will lead to the creation of aplurality of search documents 1602, a metadata index 1604, and a contentindex 1606. All of the documents are added to a search hive 1608 whichincludes a metadata store 1610 and a plurality of content stores 1612.

FIG. 17 depicts the structured data indexing process. Input data 1700 isassociated with one or more search documents 1702 in a step 1701. In theexample shown, each “processitem” is associated with its own searchdocument. Data type may also be associated with the search document in astep 1703. The search document 1702 is then parsed into different databased on information in the search document 1702 in a step 1704. Threeexamples of parsing a search document are depicted in 1706, 1708, and1710. The search document original input text is also associated withthe search document, the search document is indexed, and the results areplaced in either the content or metadata store in a step 1712. A portionof the original document as parsed is depicted in 1714.

The present invention deals with a significant amount of structuredinformation—data that has an explicit schema and is represented in awell defined format. The search engine indexing process has beenoptimized so that search queries can ask specific questions about thestructured information to obtain a more relevant set of results. Byproviding an index that is optimized around this structured informationit is possible to respond to queries that would normally have to behandled by different technologies, such as a relational database.Indexing within the system is performed as follows. If the input data isunstructured, its contents are processed according to general indexingrules defined within a search engine such as the Lucene search engine.The results are placed within an index that is then associated witheither the metadata store or the content store, depending on ruledefined within the system for the data object. If the input data isstructured, the XML is parsed and individual records are extracted forindexing. One or more search documents are created (the scheme can varybased on the goal of index optimization; in some instances a searchdocument may be created for each individual record contained in the datato be indexed). For each individual record, information representing the“type” of the input data is applied to the search document. For example,one embodiment of the invention uses Multipurpose Internet MailExtensions (MIME) to represent type information. The content of theindividual record is parsed and an alternate representation may becreated and placed in the search document to facilitate richer searchquery capabilities. This applies to records that have one or morefield/value items within them (e.g. “port: 22”, “pid: 967”).

In one embodiment, a system according to the present invention may applythree separate indexing enhancement schemes. In the first scheme, XML isremoved from the record and a plain representation of “field:value” isadded to the search document. For example, “<pid>967</pid><port>22</port>” would become “pid:967” and “port:22”. In the secondscheme, the XML is removed from the record and a plain representation of“field value field value” is added to the search document. Continuingthe example from above, “pid 967 port 22” would be a sample of this formof representation. In the third scheme, the XML may be removed and thefield and value are added to the search document in the same “location.”Location concepts within search documents are most commonly used forfuzzy matching—that is, configuring an index to respond to “similar”search queries as opposed to exact matches (e.g., if a user queries for“the quick brown fox”, documents that contain “the fast brown fox” mightalso be returned). This same concept can be applied to enhance searchqueries that are trying to find explicit values associated with specificfields.

To complete the indexing, the original, unmodified record with full XMLrepresentation may be added to the search document. The search documentis then indexed, and its results placed within an index that is thenassociated with either the metadata store or the content store,depending on rule defined within the system for the data object.

The method outlined above for addressing structured XML data can beaccomplished by the search service even if schema is not present. Thedata need only be a well-formed XML document in order for search to“guess” at the proper structure for field/value pairs. If schema is alsopresent, then stronger typing information about individual fields can beadded to the shared document as it parses individual records.

FIG. 18 illustrates one embodiment of the indexing process. Input data1802 is placed into search documents 1804 and indexed according to anindex strategy 1806 into temporary indexes 1808 in a step 1800. Fulltemporary indexes in a temporary index queue 1812 are merged accordingto an index strategy 1814 into permanent indexes 1816 in a step 1810.When a permanent index 1816 becomes full, the next temporary index inthe temporary index queue 1812 is promoted into a permanent index 1816in a step 1818.

There are several performance challenges when managing the process ofindexing. In general, it is less burdensome to add new items to smallindex and more burdensome to add items to a larger index. However, it isoften more burdensome for a search engine to look up results acrossmultiple smaller indexes than it is for a search engine to look upresults in a single large index. To balance the impact of thesecharacteristics, the present invention implements an index managementmethod that optimizes the performance of creating new indexes from inputdata, and then amortizes the cost of merging those indexes into fewer,larger indexes to increase search engine performance. The presentinvention also tracks unused indexes so that an index that is not usedby the search engine to resolve queries for some predetermined period oftime or number of total queries processed, or a combination of the two,may be removed to free up additional system resources for other uses.

The process of index management is performed within the search serviceon the controller as follows. This process is illustrated in FIG. 18.First, as input data is indexed, document content is indexed into atemporary index according to an index strategy. The index strategy is aset of parameters that governs how a temporary index may be divided andprocessed. Examples of parameters and criteria for dividing an indexinclude, but are not limited to the number of rows from a structured setof input data (e.g., record oriented XML) that should be contained in asingle search document, the number of search documents that should becreated in memory before being indexed and written to disk, and/or thenumber of search documents that should be indexed within a singletemporary index before a new temporary index is created.

The input data may be indexed into an existing temporary index if thatindex is not already at a threshold that calls for the creation of a newtemporary index. If a threshold is crossed, a new temporary index iscreated and the input data indexed into it.

Second, if a new temporary index was created, it is added to a queue oftemporary indexes that have been made available to the search engine.The search engine is able to search across all temporary indexes and allpermanent indexes.

Third, an index management process reviews the queue of temporaryindexes, “grabs” the index at the front of the queue, and begins tomerge the temporary index (presumably a small index, optimized for fastindexing) into a permanent index (presumably a larger index optimizedfor fast lookups). Both the temporary index being processed and thepermanent index being merged into are available to the search engine forqueries during this time.

Fourth, if the permanent index crosses a defined threshold defined inits Index Strategy, it is closed for writing/merging. The next temporaryindex in the processing queue is converted into a permanent index (aprocess that simply involves changing the definition of the temporaryindex; none of its contents need to be processed during thisconversion), and is then used for subsequent temporary index merges.

The present invention provides services for analysis of data within thesystem once it has been created or acquired. Analysis is a seriesoperations performed on existing data (inputs) which produces anotherset of data that may be comprised of elements or derivatives of theoriginal inputs. The present invention provides a method for users tospecify inputs, select analytical operations to perform on those inputs,specify parameters for those analytical operations, and receive and viewresults.

One embodiment of the invention provides this capability through adedicated service which functions as a component of the controller. Theanalysis service is able to communicate directly with all elements thatcomprise the controller, including those that process jobs received fromusers, interact with deployed agents, store data collected from agents,index and search data within the system, and “virtualize” documentswithin the system into relational tables. The analysis service mayutilize any of these in combination to perform operations requested bythe end user.

Individual analysis modules, called analyzers, provide the logicnecessary to implement one or more analysis commands. Analyzers arebuilt on top of an extensible framework, allowing additional analyzersto be written based on a definition of a desired analysis result. Thefunction of the analyzer is to define a series of required inputs,perform any necessary transformations on those inputs to organize themfor analysis, use a core set of analysis functions to further transformor organize the data, use a set of custom analysis functions definedwithin the analyzer, and perform a final transformation into one or moreoutput documents. Each step is described in more detail below:

Analyzers define and retrieve inputs. Analyzers may specify a number andtype of inputs. One input will always be a data object within thesystem. Inputs may also be a set of arguments and parameters thatfurther describe operations to be performed on the input. For example,for a time skew analyzer, the inputs may be one or more documents thatcontain time data along with an argument indicating how much time toskew each document by. Once inputs are defined, the Analyzer retrievesthem and moves to the next step of the analysis.

Analyzers transform inputs. In order to perform certain analyses, inputsmay need to be transformed—their representation may need to change inorder for additional analysis steps to be performed. In some situationsthe transformation of the inputs may be the analysis in its entirety,with no additional steps in the process required except production ofoutput. Most analyses will require some form of input transformation.Transformations could be a simple restructuring of the XML documentcontaining an input (e.g. extracting data from one XML document tocreate another XML document that adheres to a different schema). Atransformation may also involve changing the storage representation ofan input so that a different technology can be applied in subsequentanalysis steps. For example, if an analysis calls for relationaloperations, any XML document inputs will likely have to be parsed andloaded into one or more relational database tables.

Analyzers may perform core analysis. The analysis service provides aseries of core functions that any analyzer may use to conduct analyses.These core functions are fundamental operations that are common acrossmany different classes of analysis problems, several of which mayrequire relational representations of the inputs in order to beperformed. The following fundamental operations may be performed: union,intersection, difference, and equality. Note these are not exclusive;other core functions could be defined and included in the core of theanalysis service. A union analyzes two sets of data X and Y, the unionof X and Y contains all the data in X, all the data in Y, but nothingelse. An intersection analyzes two sets of data X and Y, and creates adata set containing the intersection of X and Y that is all the data inX that is also in Y. A difference analyzes two sets of data, X and Y,and creates a data set containing the difference of Y and X which is theset of data in Y but not in X. An equality analyzes two sets of data Xand Y, and creates a data set containing data contained in set X and setY and all data in set Y is also in set X.

Analyzers may perform custom analysis. Some analyzers may not be able toaccomplish their objective through applying transformations and usingcore analysis services. In those instances, analyzers may directlyimplement the analysis logic themselves, using services within thesystem (such as virtualized data—that is, data stored in a relationaldatabase) to accomplish this goal. Any capabilities required by theanalyzer that are not implemented by the analysis service are fullyimplemented within the analyzer itself

Analyzers transform outputs and write results. Once inputtransformations and core analysis is complete, the results of theanalysis may need additional transformation, similar to inputtransformation above, before they can be written back into the system.For example, any results that are in a relational database table aretransformed back into an XML representation. Results may also betransformed into a format required by a user. Once all results have beenwritten into representations understood by the remainder of the system(e.g., XML documents), the analysis is complete.

FIG. 19 illustrates the analysis service, the concept of modularanalyzers, and the flow of an analysis through input definition, inputtransformation, core analysis, custom analysis, and output generation.The process depicted in FIG. 19 encompasses defining and retrievinginputs 1902, transforming inputs 1904, core analysis 1906, and customanalysis and recording of results 1908. A job document is received by ananalysis service 1912 in a step 1914. The analysis service 1912retrieves the input data specified by the job document 1910 in a step1916 which may be process list A and process list B from data managementservices 1918. Data is transformed from its XML document representationinto a relational database table in a step 1920. The difference coreanalysis function is used within the relational database holding theprocess list tables, resulting a left difference and right differenceresult table in a step 1922. The left and right difference tables aretransformed into XML documents adhering to appropriate schema in a step1924. The collection of result documents is referenced by a job resultdocument. The results can now be written to the system using datamanagement services.

A key element of the invention is the ability for multiple users tointeract with the system at the same time. Given a set of controllersarranged as a cluster, users interacting with those controllers througha console are able to utilize all aspects of the system in a fashionthat facilitates sharing of data, allows identification andunderstanding of changes and modifications made by other users, andprevents collisions or contention between resources that result inunexpected or non-deterministic changes within the system.

The fundamental functions of the present invention support thiscollaborative paradigm in the following fashion. For security purposes,users within the system may be identified in the system through the useof a unique identifier. The user, in their role as a trust domain entity(discussed above), directly authenticates into the controller cluster.Every operation undertaken by the user is marked in audit records withinthe controller such that all creation of new data or changes to existingdata can be associated with a given user.

Data is acquired through the agent. Users may launch jobs to acquiredata from agents simultaneously. The controller job management methodsprovides queuing, scheduling, and binding of data results to specificjob request such that multiple requests from multiple users to the sameagent will result in separate sets of acquired data responsive to eachuser's request.

Much like data acquisition, multiple users may conduct analysissimultaneously. Controller job scheduling results in analysis resourcesbeing shared across jobs as they are received by the controller. Notethat a job to conduct analysis is self-contained—all of the data inputsnecessary to conduct the analysis must either already exist, or must beproduced by initial steps contained within the analysis job itself. Thisis in keeping with the stateless paradigms discussed in the section ondata representation and access.

The present invention follows stateless paradigms as discussed above inconnection with data representation and access. As such, any time datais modified (this includes the creation and deletion of data), the“current” state of the data reflects the state after the lastmodification made. Each request for modification is fully self-containedand does not require multiple transactions from several requests from amodifier (e.g., a single request or “transaction” from the console tothe controller contains all the information necessary to perform amodification). Each update operation is, in essence, atomic—that is tosay the series of actions necessary to modify data within the systemappear to the rest of the system as a single operation with only twopossible outcomes: success or failure.

The present invention also supports state updates. While the presentinvention is in essence stateless, and follows REST-based methods forsoftware architecture as discussed above, it does provide somefacilities for clients (e.g., the console) to portray a more detailedstate picture for the end user. The controller and all of its associatedsubsystems publish a stream of information about changes within thesystem. The system utilizes syndication feed technology such as Atom topublish a feed of changes and updates within the system. In keeping withREST software methods, the controller does not infer any complex statebased on this stream of updates. Instead, a client must consume the feedand make its own determinations on how it wishes to use those to updatea user's view of system state.

The present invention provides for searching within the system ispossible by multiple users at the same time. Additionally, the searchengine is responsive while indexing operations are ongoing. Usersreceive results for indexes that are currently available to the system.If a document is partially indexed, those partial results are availableto end users.

Given these considerations, the present invention provides severalfacilities to further allow users to collaborate when collecting,analyzing, or presenting information during the course of an EED orcomputer security incident. These include data markup and organization,reporting, and detailed auditing.

A common method to perform data collection, analysis, and minimizationis to divide tasks among available resources (e.g., investigators oranalysts), and to collate results centrally. Investigators and analystsmay be more familiar with certain sections of acquired or analyzed data;it is common to have someone cross-check or re-check findings based ontheir expertise or focus.

Working in this fashion requires several different methods of dataorganization. Users may wish to organize all findings for a particularcomputer security or EED incident into a case—a logical organization ofrelated information pertaining to the events at hand (e.g., “a casefile”). Users may also wish to perform more ad-hoc organizations,marking data according to any number of factors. Users may wish toidentify elements of an ad hoc workflow associated with data,conclusions reached during review, relevance, or any other number offactors. In fact, one of the primary challenges in a system addressingthis form of problem is the incredible variance of ways end users wishto organize and mark information.

To that end, the invention provides facilities for arbitrary markup ofacquired and analyzed data. The system implements the concept ofattributes. An attribute is additional data that can be attached to anyentity within the system. Attributes typically store small amounts ofdata, such as a label (e.g., “evil”, “evidence”, “mark's work”). When anattribute is de-referenced (examined to identify all entities it isattached to), sets of data that the attribute was applied to arereturned to the requestor. This provides a powerful method for bothstructured and ad-hoc data organization within the system.

One embodiment of the present invention provides three primary attributeconcepts: labels, properties, and notes. Other attribute concepts arepossible provided they meet the primary requirement of being data thatcan be attached to any entity within the system.

Labels are typically text strings that allow the end user to “label” anentity (e.g. a row of data within an audit returned from an agent, adocument within the system, a group of documents that are related toeach other). Labels are typically short text strings (though they couldbe “long” if so desired) that provide organizational or status cues tothe end user. Examples would include “case” labels (e.g., “Aug. 16,2007: Fraud Case”), work status (“reviewed”, “completed”), andanalytical conclusions (“evil”, “malware”, “clean”). Users could applylabels in a variety of ways tailored to their particular workflow orprocess.

Properties are labels with an associated value (e.g., “complete=10%”).They provide a method to communicate a more discrete set of informationthan a label, while still carrying all of the characteristics of anattribute within the system—it can be applied to any entity, andde-referencing it produces a set of data that the property is attachedto.

Notes are simply free-form text. They could be thought of as a “long”label, but might be used to contain more detailed information about theobject they are attached to. Notes are typically more descriptive thanlabels, with their primary purpose being more content oriented thanorganizational.

Attributes are data objects within the system, just like other entitiessuch as Documents and Audit Results. Attributes share the same RESTproperties as other data types—they are directly addressable via a URI,their data is formatted and stored using XML according to a specificschema, and all operations conducted against an attribute (creation,deletion, or modifying it such that it is attached or “applied” toanother data object) are stateless, atomic transactions.

Attributes store a number of data elements that describe both their owncontent (e.g., their name, their identity), and the objects they areapplied to (e.g., a list of URIs to other objects in the system). Thelist of objects an attribute is applied to is, in essence, a containerand is represented using a syndication feed, such as an Atom feed, whilebeing stored in a relational database table.

In order to support the concept of “attributing” individual row itemswithin a document—for example, a series of file items within a filelisting audit—an additional data object exists within the system calleda markup. Markups store information about rows within a document thathave a particular attribute applied to them. There are two possiblemethods for utilizing markups for row item attribution within thesystem: explicit use of markup as a directly attributed object, andone-to-one markup-to-object mappings such that one markup contains allrow item attribution information for a single corresponding data object.

In a directly attributable markup, the markup data object storesinformation about a group of rows within a document. It contains theidentity of the document it is applied to and information about a set ofrows within that document. The markup may then have an attribute appliedto it.

In a one-to-one markup, the markup is directly associated with a dataobject. The data object includes the URI of its corresponding markupobject. The markup object contains information about all rows within thedata object that are grouped as a “set”, and it also identifies the URIof the attribute that is applied to those rows.

An embodiment of the invention will typically select one method to useglobally for row item attribution throughout the system—either directlyattributable markup or one-to-one markup.

The presence of an attribute on an object or series of row items isimportant information that users may wish to have displayed in a numberof different contexts, depending on the user's purpose for theattribute. The context of the attribute must be understood in order tounderstand the most useful method to utilize when presenting it to auser. It may not be possible to determine the user's desiredpresentation for a given attribute at the time it is created. Therefore,the present invention provides two primary methods for presentingattribute information to end users, allowing the user to choose the mostrelevant context according to their needs.

In one implementation of the invention, labels are the primary attributecapability provided to users. Labels may be used in one of two contexts:organizational or as virtual data for a data object.

As discussed above, when an attribute such as a label is de-referenced(that is, examined to view its contents), it provides a list of objectsin the system that it is applied to. This is analogous to the concept ofa “folder”, or other form of data container on a computer system—forexample, a “directory” on a computer file system provides a similarcapability. The present invention provides a view into the system thatpresents a list of labels that the user can select. When the use selectsa label, the contents of that label—the list of objects the label isapplied to—is rendered for the end user. FIG. 20 illustrates onepossible method for visualizing labels 2010 and the data they areapplied to 2020 in an organizational context.

Labels may also be virtual data. Labels may sometimes be used to addinformation to an object. The user wants to be reminded about that extrainformation when they view the data item. In one embodiment of theinvention, the concept of virtual columns is used. The data object isdisplayed to the user; if a label has been applied to the object, anelement is added to the display that shows the label has been applied tothat object. When the view into the object is in a grid (much like aspreadsheet), a column is added to the display that contains thisinformation. In the case where multiple data items are being viewed, allobjects in that view containing the label are appropriately marked, withthe name of the label serving as the column header. FIG. 21 illustratesone possible method for visualizing labels and the data they are appliedto in a virtual data context.

During the investigative process for either electronic discovery or acomputer security incident, investigators and analysts need to recordtheir results and observations. Such records are often collected in theform of a document or series of documents, and include snippets ofacquired data, analytical results, and the conclusions of theinvestigator based on their personal experience and observation. Thepresent invention provides a method for users to record this data withindocuments and do so in a fashion that allows for collaborative editingand sharing of these records.

Report documents may link to data within the system. In one embodiment,users may create Extensible Hypertext Markup Language (XHTML) baseddocuments. The system refers to these as case notes; however, they aresimply documents that permit semi-arbitrary input of user content. Userscan directly edit content within the document by using the consoleportion of the invention. In addition to typical word-processing/HTMLediting features, users can insert hyperlinks to any data object withinthe system—since the entire system is REST based and all objects have aURI, any of those URIs can be embedded within the body of a case notesdocument. When links are embedded in the document, they can either beviewed as a standard hyperlink, or expanded to represent a portion orthe entire object they reference based on the user's preference. If auser clicks on those links, the console retrieves the content of thatlink and displays it to the end user, much like a web browser. FIG. 22illustrates this concept.

Any user of the system with access to a given case notes document canopen it for editing. A unit of collaboration (UC) is the smallest unitof content which may be edited by more than one user at a time. When theuser saves (or “commits”) the case notes document, the collaborationengine (CE) tests whether the authority UC (the UC in storage on thecontroller) is newer than the working UC (the UC the end user is editingand wants to commit). If not, the commit continues (the working UC isstrictly newer than the authority UC). Otherwise, the commit fails andthe user is prompted for action, given a means to update the working UCto the authority UC (losing their changes), commit their copy anyway(overwriting the current authority UC), or save the working UC to adifferent document for manual merging later (its content is marked as aderivative of the authority UC to simplify merging later). Oneembodiment of the invention places the collaboration engine in theconsole; however, it could also be implemented within the controller.

Another embodiment of the invention can provide a more advanced methodfor collaborative editing that utilizes versioning. Every commit of adocument is a revision and older revisions may be accessed individually.Deletion is a revision which marks the resource deleted so that previousrevisions may still be retrieved. Conflict resolution is as above butenhanced by the ability to “optimistically” commit the working unit ofcollaboration immediately, persisting changes, then retrieve a previousrevision to merge (if necessary). Further, the cost of an erroneouscommit is reduced by the ability to manually (or automatically) rollback to a previous revision. These methods are similar to those used bysource code control systems such as Subversion.

To further reduce the occurrence of such conflicts, a case notesdocument is a master document containing one or more sections, each ofwhich is a unit of collaboration. The scope of a user's edits andcommits are across a single section at a time, making it possible formultiple users to simultaneously edit a single document made up ofmultiple sections.

One implementation of this model would use a syndication feed ofentries. The entries in the feed reference their source documents viarelationship links; none contain content (although they may containsummary data which provides a read-only view into the target's currentcontent). When rendering a document (such as case notes), all sectionsare retrieved individually and asynchronously. Read-only feed consumers(such as supervisors or off-unit audit services) may retrieve the datain its native syndication feed format.

Within a document, a section is any new titled content. A section is acollaboration atom, as above. Edit locking is maintained on the clientso that the section the user is “editing” (where the user's input cursoris) is locked for updates but other sections can be updated live whilethe user is working in the document. If no changes have been made to asection, it becomes unlocked for updates as the user exits it and entersanother section. Section generation can be transparent to the user(whenever a heading is created) or manual (similar to a threadeddiscussion where a user clicks to insert a ‘reply’ between previoussections).

All of the above is designed to provide a conservative, stateless userexperience where even disconnected work is possible. As that may yet beinadequate to provide a good user experience, a controller mechanismwhich allows clients to publish to the common audit trail may provideediting notification, similar to the controller's use of a syndicationfeed such as Atom to update consoles to changes within the system. Usingthat mechanism, an audit trail event is posted by a client when a userbegins editing a unit of collaboration or cancels an edit (otheroperations, such as a commit, are already properly published).

Since a commit of a unit of collaboration is reported via the audittrail, the user may be immediately notified (with a non-modal statuschange) that the Working UC they are editing is out of date, giving themthe option to resolve via the process above.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the present invention asdefined by the claims, the foregoing description of the preferredembodiment should be taken by way of illustration rather than by way oflimitation of the invention set forth in the claims.

The invention claimed is:
 1. A method of organizing and presenting datagathered in response to an event, the method being performed by one ormore computers coupled to a network and comprising the steps of: (a)authenticating a plurality of memory sources by verifying whether acertificate presented by each of the plurality of memory sources isvalid within a trust domain, each of the certificates indicating whethera respective one of the plurality of memory sources and all stored dataof the respective one of the plurality of memory sources can be trusted;(b) in response to a successful authentication of the plurality ofmemory sources, importing the stored data from the authenticatedplurality of memory sources into a first memory coupled to the network;(c) converting the stored data into a specified format to produceuniform data, the specified format being one of general purpose markuplanguage and plain text format, the stored data providing a relationalcomparison between sets of the stored data; (d) providing an interfaceto a user through which the user requests a search of the uniform data;and (e) presenting one or more subsets of the uniform data to the userin response to the request from the user, wherein, the event is one of(i) a request for production of documents, and (ii) a data securitybreach.
 2. The method of claim 1 wherein the plurality of memory sourcesare coupled to the first memory through the network.
 3. The method ofclaim wherein the plurality of memory sources are servers.
 4. The methodof claim 1 wherein the uniform data is stored in a database prior to therequest from the user.
 5. The method of claim wherein the stored datacomprises both content and metadata.
 6. The method of claim 1 whereinthe interface is provided to a plurality of users through the network.7. The method of claim 1 wherein the request for production of documentsis in a legal process.
 8. The method of claim 1, further comprising thestep of: creating a certificate authority in response to the event, thecertificate authority (i) configured for use by the trust domain, and(ii) operable to create and sign each of the certificates.
 9. The methodof claim 8, wherein the verification is based at least in part on acertificate revocation list published by the certificate authority. 10.The method of claim 1, wherein the stored data is described in a schemaform for the relational comparison between the sets of the stored data.11. The method of claim 10, wherein the schema form is a description ofthe data using a syntax.
 12. The method of claim 1, wherein theplurality of memory sources are disk drives.
 13. The method of claim 1,further comprising the step of: installing an agent on the at least onememory source of the plurality of memory sources, the agent configuredto connect to an agent discovery service.
 14. The method of claim 13,wherein the agent is configured to transmit the certificate to the agentdiscovery service.
 15. The method of claim 1, wherein the first memoryis a form of removable media operable to permit installation of thestored data on a system.
 16. The method of claim 15, wherein the form ofremovable media is a USB memory storage key, an externally-connectedhard drive, or a floppy disk.
 17. A method of organizing and presentingdata gathered in response to an event, the method being performed by oneor more computers coupled to a network and comprising the steps of: (a)installing an agent on at least one of a plurality of memory sources;(b) presenting a certificate from the at least one of the plurality ofmemory sources via the agent; (c) authenticating the at least one of theplurality of memory sources by verifying whether the certificatepresented by the at least one of the plurality of memory sources isvalid within a trust domain, the certificate indicating whether storeddata of the at least one of the plurality of memory sources can betrusted; (d) in response to a successful authentication of the at leastone of the plurality of memory sources, importing the stored data fromthe at least one of the plurality of memory sources into a first memorycoupled to the network; (e) converting the stored data into a specifiedformat to produce uniform data; (f) providing, via a computer coupled tothe network, an interface to a user through which the user requests asearch of the uniform data; and (g) presenting one or more subsets ofthe uniform data to the user in response to the request from the user,wherein, the event is one of (i) a request for production of documents,and (ii) a data security breach.
 18. The method of claim 17, wherein theplurality of memory sources are coupled to the first memory through thenetwork.
 19. The method of claim 17, wherein the plurality of memorysources are servers.
 20. The method of claim 17, wherein the specifiedformat is one of general purpose markup language and plain text format.21. The method of claim 17, wherein the stored data providing arelational comparison between sets of the stored data.
 22. The method ofclaim 17, wherein the uniform data is stored in a database prior to therequest from the user.
 23. The method of claim 17, wherein the storeddata comprises both content and metadata.
 24. The method of claim 17,wherein the interface is provided to a plurality of users through thenetwork.
 25. The method of claim 17, wherein the request for productionof documents is in a legal process.
 26. The method of claim 17, furthercomprising the step of: creating a certificate authority in response tothe event, the certificate authority (i) configured for use by the trustdomain, and (ii) operable to create and sign the certificate.
 27. Themethod of claim 26, wherein the verification is based at least in parton a certificate revocation list published by the certificate authority.28. The method of claim 17, wherein the stored data provides arelational comparison of data.
 29. The method of claim 17, wherein thestored data is provided in a schema form with a description of the datausing a syntax.
 30. The method of claim 17, wherein the agent isconfigured to connect to an agent discovery service.
 31. The method ofclaim 30, wherein the agent is configured to transmit the certificate tothe agent discovery service.
 32. The method of claim 17, wherein thefirst memory is a form of removable media operable to permit theinstallation of the stored data on a system.
 33. The method of claim 32,wherein the form of removable media is a USB memory storage key, anexternally-connected hard drive, or a floppy disk.